Behavior-Based Quality Assurance in Crowdsourcing Markets
نویسندگان
چکیده
Quality assurance in crowdsourcing markets has appeared to be an acute problem over the last years. We propose a quality control method inspired by Statistical Process Control (SPC), commonly used to control output quality in production processes and characterized by relying on time-series data. Behavioral traces of users may play a key role in evaluating the performance of work done on crowdsourcing platforms. Therefore, in our experiment we explore fifteen behavioral traces for their ability to recognize the drop in work quality. Preliminary results indicate that our method has a high potential for real-time detection and signaling a drop in work quality.
منابع مشابه
Worker Perception of Quality Assurance Mechanisms in Crowdsourcing and Human Computation Markets
Many human computation systems utilize crowdsourcing marketplaces to recruit workers. Because of the open nature of these marketplaces, requesters need to use appropriate quality assurance mechanisms to guarantee high quality results. Previous research has mostly focused on the statistical aspects of quality assurance. Instead, we analyze the worker perception of five quality assurance mechanis...
متن کاملMentor: A Visualization and Quality Assurance Framework for Crowd-Sourced Data Generation
Crowdsourcing is a feasible method for collecting labeled datasets for training and evaluating machine learning models. Compared to the expensive process of generating labeled datasets using dedicated trained judges, the low cost of data generation in crowdsourcing environments enables researchers and practitioners to collect significantly larger amounts of data for the same cost. However, crow...
متن کاملTowards Successful Crowdsourcing Projects: Evaluating the Implementation of Governance Mechanisms
The last decade has witnessed the proliferation of crowdsourcing in various academic domains including strategic management, computer science, or IS research. Numerous companies have drawn on this concept and leveraged the wisdom of crowds for various purposes. However, not all crowdsourcing projects turn out to be a striking success. Hence, research and practice are on the lookout for the main...
متن کاملProgrammatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing
Crowdsourcing is an effective tool for scalable data annotation in both research and enterprise contexts. Due to crowdsourcing’s open participation model, quality assurance is critical to the success of any project. Present methods rely on EM-style post-processing or manual annotation of large gold standard sets. In this paper we present an automated quality assurance process that is inexpensiv...
متن کاملData Analytics on Online Labor Markets: Opportunities and Challenges
The data-driven economy has led to a significant shortage of data scientists. To address this shortage, this study explores the prospects of outsourcing data analysis tasks to freelancers available on online labor markets (OLMs) by identifying the essential factors for this endeavor. Specifically, we explore the skills required from freelancers, collect information about the skills present on m...
متن کامل